fix(#1157): NemotronH decoded with STALE input ids, and the recurrent carry was never the cause - #1221

…DING because I read a body written before its own job ran

The previous commit here wrote the A3 end-to-end token gate as `PENDING` and
said no GB10 96/96 existed. That was wrong, and the way it was wrong is worth
keeping: I searched `origin/main` and five `row/MODEL-NEMOTRON-H-*` branches,
found nothing, and then took #1221's PULL REQUEST BODY as the current state. Its
"Still owed: the sm_121a green-after re-run" was TRUE WHEN WRITTEN and stale by
the time I read it. A body is a snapshot of its author's knowledge at write time,
not a live field, and no amount of branch searching corrects for that, because
the artifact was never in git at all.

It is on the NAS. `/usr/local/nas_share/rc/nh1157/gate_fixed.out`:

    TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
    STRICT PASS

against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on
`nemotron-3.5-lightning-30b-nvfp4` at revision `29f2d1746d8f41e316523194b19018707749b1b1`.
Verified by reading the files, not by trusting the report of them.

It is the DEVICE leg. The binary is `libvllm 0.0.3+cuda` and `cfg.log` records
`fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` `ENABLED for [121a]`. The run logs
`Asynchronous scheduling is enabled (max_concurrent_batches=2)`, which is exactly
where `device_token_ids` is non-null; on the host queue it is always null and the
#1157 defect cannot arise. The decisive one is neither: the same binary on the
same checkpoint with ONLY `nemotron_h_device.cpp` reverted to the fix's parent
scores `4/24 (full rows=0, short rows=3)` and bails at 8 tokens. A host-leg run
would have been unmoved by that revert. The delta is the proof; the pass alone
is not.

So the row now says the gate PASSES and says whose it is. It belongs to #1221,
branch `row/MODEL-NEMOTRON-H-ABI-A2P-1157-fix` at `6e9e8955`, OPEN and
`CONFLICTING`. `main`'s last touch of `nemotron_h_device.cpp` is `a6df72777`
(A2-P), so `main` does not carry the fix and is not gated. A gate that passed and
a tree that is gated are different facts, and this record keeps them apart.

No performance number is claimed anywhere. `gate_fixed.out` carries 264.4s to
load and 327-343s per 32-token prompt; that is a correctness run whose `lm_head`
and 46 FP8 mamba projections still execute host-side. Both public rows say so in
the words "NOT a benchmark", so nothing can later lift them into a grid. One
config caveat also travels: `--gpu-memory-utilization 0.92` did not size the KV
pool, which fell back to 256 blocks (#83).

The row stays `PARTIAL`. `ACTIVE` still fails the `CLAIM-*` rule, and a gate that
passes off `main` is not a reason to move a lifecycle state.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

The explicit merge of `row/MODEL-NEMOTRON-H-ABI-A2P-1157-fix` this branch
carried was a stand-in for a fix that had not landed. It has: `0ea5d249f` is an
ancestor of `origin/main`, so the measured base is now main itself and the diff
collapses to this branch's own work.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

# Conflicts:
#	examples/nemotron_h_gen/main.cpp

…overs one configuration rather than a model (#1250)

#1221 landed as `0ea5d249f`, verified by content on `main` rather than by the
API, so the spec's stand-in wording about measuring off the fix branch is dead
and the measured base is a SHA at or after that merge.

The second edit is the one that matters. A token gate is evidence about the
configuration it ran, and this one ran three prompts, 32 greedy tokens each,
batch 1 sequential, `max_model_len 512`, a 256-block pool. The timing legs run
that configuration and nothing else, and a leg that deviates in KV sizing,
batching or sampling is reported as ungated rather than allowed to borrow
coverage the gate never gave it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` gained #1221 while this branch was open, so `docs/BENCHMARKS.md` and
`docs/STATUS.md` conflicted on the NemotronH rows. Resolved by taking main's
rows as the base and correcting the one claim that main is now STALE on.

Main says "GB10 read 4/24 ... sm_121a re-run pending a lease" and "sm_121a
re-run pending". That was true when #1221 was written. The re-run has since
happened: the A3 gate PASSES on GB10 at 96/96, `STRICT PASS`, mode=decode,
against `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4`
revision `29f2d1746d8f41e316523194b19018707749b1b1`, with a red-before of 4/24
on the SAME binary and checkpoint with only `nemotron_h_device.cpp` reverted.
Artefacts: `/usr/local/nas_share/rc/nh1157/{gate_fixed,gate_red,cfg}.out`.
`cfg.log` shows `fp4-mma`, `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and
`fa2` all `ENABLED for [121a]`, so the build was not degraded, and both legs log
`Asynchronous scheduling is enabled`, which is the path where
`device_token_ids` is live and the defect lived.

The delta, not the pass, is the proof: on the host leg reverting the fix would
have changed nothing.

STILL NOT A BENCHMARK. The 264.4 s load and 43 405 MB peak are recorded as a
correctness run and labelled as such, because NVFP4 `lm_head` (A2-Q2b) and the
46 FP8 W8A8 mamba projections (A2-Q1, #940) still execute host-side. No
throughput, latency or memory ratio is claimed on any axis.

The index was reconciled by discard-and-reapply: main's file taken wholesale,
only rows whose key main lacks re-appended, main's file a strict PREFIX.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…and the GPU is idle for 93.7% of the decode (#1250, #1253)

Numerator only, and no ratio is claimed. On an idle `dgx:gpu0` through `rc run`,
at a tree containing #1221, with all five CUDA feature lines `ENABLED for
[121a]`: engine load 280.9 s, 10.3194 s per output token, 0.09691 tok/s at batch
1, peak host 44,616 MB of 122,502. Every timing leg is a gated leg -- 96/96
`mode=decode` on both, `STRICT PASS`, 192 of 192 tokens, zero short rows -- so
no number here comes from a configuration whose tokens were not compared in the
same process. The same-binary A/B over one load is 1.0016 warm and the five warm
prompts spread 0.245%, so the figure is stable enough that the next lever's
delta will be readable.

The load-bearing part is not the rate, it is why. `nvidia-smi` reported GPU
utilization 0% in 2,019 of 2,155 samples across the measured window, so the GPU
was busy in 6.31% of it, and `gpu_clock_state`'s own comparison gate would
REFUSE that window for failing its majority-busy floor. The refusal is the
result: this decode is not GPU work. It is the 23 of 52 layers that download the
normed hidden, run the mixer on the CPU queue and upload it once per layer per
token, plus the NVFP4 `lm_head` projecting on the host. So A2-Q1 (#940) and then
A2-Q2b are the next traceable hypotheses, and each now has a prediction it has
to answer rather than a story: it must raise that busy fraction.

The denominator has its identity pinned and its blocker measured rather than
assumed. With `python3-dev` present the pinned oracle initialized, loaded in
230.4 s and completed `torch.compile`, which is further than it has ever got
inside a lease, then took 104,992 MB of host in the next step and was killed by
this row's watchdog. The box did not reboot. `gpu_memory_utilization` 0.9 of
~119 GiB of unified memory is ~107 GB, which is that peak, so the fraction is
the first thing to vary. And vLLM's hybrid allocator forces a 4192-token
attention block against our 32, so the two sides can be matched on KV token
capacity and never on block count; a future ratio has to say which.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…DING because I read a body written before its own job ran

The previous commit here wrote the A3 end-to-end token gate as `PENDING` and
said no GB10 96/96 existed. That was wrong, and the way it was wrong is worth
keeping: I searched `origin/main` and five `row/MODEL-NEMOTRON-H-*` branches,
found nothing, and then took #1221's PULL REQUEST BODY as the current state. Its
"Still owed: the sm_121a green-after re-run" was TRUE WHEN WRITTEN and stale by
the time I read it. A body is a snapshot of its author's knowledge at write time,
not a live field, and no amount of branch searching corrects for that, because
the artifact was never in git at all.

It is on the NAS. `/usr/local/nas_share/rc/nh1157/gate_fixed.out`:

    TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
    STRICT PASS

against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on
`nemotron-3.5-lightning-30b-nvfp4` at revision `29f2d1746d8f41e316523194b19018707749b1b1`.
Verified by reading the files, not by trusting the report of them.

It is the DEVICE leg. The binary is `libvllm 0.0.3+cuda` and `cfg.log` records
`fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` `ENABLED for [121a]`. The run logs
`Asynchronous scheduling is enabled (max_concurrent_batches=2)`, which is exactly
where `device_token_ids` is non-null; on the host queue it is always null and the
#1157 defect cannot arise. The decisive one is neither: the same binary on the
same checkpoint with ONLY `nemotron_h_device.cpp` reverted to the fix's parent
scores `4/24 (full rows=0, short rows=3)` and bails at 8 tokens. A host-leg run
would have been unmoved by that revert. The delta is the proof; the pass alone
is not.

So the row now says the gate PASSES and says whose it is. It belongs to #1221,
branch `row/MODEL-NEMOTRON-H-ABI-A2P-1157-fix` at `6e9e8955`, OPEN and
`CONFLICTING`. `main`'s last touch of `nemotron_h_device.cpp` is `a6df72777`
(A2-P), so `main` does not carry the fix and is not gated. A gate that passed and
a tree that is gated are different facts, and this record keeps them apart.

No performance number is claimed anywhere. `gate_fixed.out` carries 264.4s to
load and 327-343s per 32-token prompt; that is a correctness run whose `lm_head`
and 46 FP8 mamba projections still execute host-side. Both public rows say so in
the words "NOT a benchmark", so nothing can later lift them into a grid. One
config caveat also travels: `--gpu-memory-utilization 0.92` did not size the KV
pool, which fell back to 256 blocks (#83).

The row stays `PARTIAL`. `ACTIVE` still fails the `CLAIM-*` rule, and a gate that
passes off `main` is not a reason to move a lifecycle state.

`docs/FEATURES.md` moves in the same commit, because it has to. This commit
edits `.agents/model-matrix.md`, which `scripts/check-doc-checkpoint.py` reads as
a `feature_surface` change by path (`FEATURE_SURFACE_FILES`), and that class owes
`docs/FEATURES.md` in the SAME commit -- the checker walks a range one commit at
a time, so no later commit can pay the debt. The row for
`NemotronHForCausalLM` said nothing at all about the end-to-end token gate while
the matrix said it was PENDING, and it would have kept saying nothing now that
the gate passes. It now carries the same two facts the matrix does, in the
correctness column and in that column's own words: 96/96 `STRICT PASS` on GB10,
on #1221's open branch and not on `main`. The added text is 90 characters, which
leaves the cell at 192 of its 220-character cap and the row at 499 of 600, so no
other row pays for it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` gained #1221 while this branch was open, so `docs/BENCHMARKS.md` and
`docs/STATUS.md` conflicted on the NemotronH rows. Resolved by taking main's
rows as the base and correcting the one claim that main is now STALE on.

Main says "GB10 read 4/24 ... sm_121a re-run pending a lease" and "sm_121a
re-run pending". That was true when #1221 was written. The re-run has since
happened: the A3 gate PASSES on GB10 at 96/96, `STRICT PASS`, mode=decode,
against `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4`
revision `29f2d1746d8f41e316523194b19018707749b1b1`, with a red-before of 4/24
on the SAME binary and checkpoint with only `nemotron_h_device.cpp` reverted.
Artefacts: `/usr/local/nas_share/rc/nh1157/{gate_fixed,gate_red,cfg}.out`.
`cfg.log` shows `fp4-mma`, `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and
`fa2` all `ENABLED for [121a]`, so the build was not degraded, and both legs log
`Asynchronous scheduling is enabled`, which is the path where
`device_token_ids` is live and the defect lived.

The delta, not the pass, is the proof: on the host leg reverting the fix would
have changed nothing.

STILL NOT A BENCHMARK. The 264.4 s load and 43 405 MB peak are recorded as a
correctness run and labelled as such, because NVFP4 `lm_head` (A2-Q2b) and the
46 FP8 W8A8 mamba projections (A2-Q1, #940) still execute host-side. No
throughput, latency or memory ratio is claimed on any axis.

The index was reconciled by discard-and-reapply: main's file taken wholesale,
only rows whose key main lacks re-appended, main's file a strict PREFIX.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…ncile rested on expired (#1074)

FOLLOWING_AGENTS_PROTOCOL

This reconcile was written while [#1221](#1221)
was OPEN, and it moved the row `INVENTORIED` -> `PARTIAL` on the strength of an
A3 gate that read 96/96 `STRICT PASS` on GB10. #1221 MERGED on 2026-08-18 as
`0ea5d249f`, which is now `main`'s last touch of `nemotron_h_device.cpp`, so
every "blocked on #1221 landing" sentence in this branch became false.

What is recorded now is narrower than what the branch originally claimed, and
deliberately so. The A3 gate read 96/96 `STRICT PASS` (mode=decode) on GB10,
MEASURED ON THE TREE THAT LANDED AS `0ea5d249f`. That fix is on `main`. **No run
against current `main` exists**, because `main` has advanced many commits past
the measured tree. Naming the SHA a measurement belongs to is the whole point:
an evidence line that names a tree it was not measured on has cost this
repository before, and this change does not add another.

**The row therefore STAYS `INVENTORIED`.** The lifecycle move is left to whoever
re-runs the gate against `main`, and the derived counts in
`.agents/model-matrix.md` are restored to main's `53`/`324`/`22` with the
engaged-architecture row withdrawn. Measured rather than asserted: the row-state
histogram over all 373 keyed rows is byte-identical to main's, so this change
moves NO row's lifecycle state, and the count block is byte-identical to main's.
What survives from the original reconcile is the part that was never about
#1221: the row's `KERNEL-SSM-MAMBA` block was FALSE, and its corrected code and
test anchors stay, so a later state move has its contract already written.

`docs/BENCHMARKS.md` carried the key `NemotronH paged forward`
(`MODEL-NEMOTRON-H-ABI-A2P`) TWICE on this branch and once on main, because the
branch APPENDED where it should have UPDATED. The two rows contradicted each
other: main's said GB10 read 4/24 with an sm_121a re-run pending a lease, and
the branch's said GB10 read 96/96. The lease and the re-run happened, so main's
row is stale on its face. They are collapsed to ONE row on the key byte-identical
to main's, carrying the current projection. Nothing gates this:
`check-public-doc-tables.py` holds the page's size and shape and not its key
uniqueness, so it would have landed silently.

`docs/STATUS.md` gets a one-line in-place edit to the `Text generation` row's
"Open gate or limitation" cell, which is where #1491 routed this after it
reorganised the page by surface. That is the shape `af25bd251`, `5702d8f83` and
`d995c52f0` established for `Speculative decoding` and the diffusion row, and it
satisfies the `lifecycle` obligation without growing a shrink-only page: 1
insertion, 1 deletion, 98 lines and 27 table rows before and after. Model-level
detail is NOT reintroduced; it stays in
[`docs/models/nemotron-3-5-lightning.md`](../docs/models/nemotron-3-5-lightning.md),
whose "no token gate result exists" line remains true of `main`.

The obligation still fires with no lifecycle move because
`check-doc-checkpoint.py` adds the `lifecycle` class for a measurement change as
well as a state change, so `STATUS` and `BENCHMARKS` are both owed and both
written.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

… open, and this branch added that text (#1074)

The `## Owed` entry for #1217 was written while #1221 was open and it kept three
claims that are false at this head. It called #1221 "open and unmerged", although
#1221 merged on 2026-08-18 as `0ea5d249f`, which the previous commit's own
subject says. It said #1217 "is recorded NOWHERE in this tree", although the
merge `8a00afcbe` brought both records in: `.agents/issue-index.md:403` carries
the index row under the owning row `MODEL-NEMOTRON-H-ABI-A2P`, and
`nemotron-h-a2p-paged-forward.md:901` lists the issue under that spec's
`## 11. Owed`. That claim was true at `b626be75a` and is not true here. And it
named #1217 "the standing reason this row's end-to-end gate is not a pass", which
contradicts `## Now` in the same file, where the A3 gate reads 96/96
`STRICT PASS`.

`git diff 6b48edb 414a800` shows this branch ADDED the phrase, and
`origin/main` carries it zero times, so the stale text is this pull request's to
remove rather than an inherited defect.

The bullet now records where the record landed and cites both anchors, which
were re-derived at this head. The duplicate-key argument for withholding an index
row is kept as history and marked expired: appending a row now would create the
very duplicate that argument was written to avoid, because the row is already on
`main`. Nothing is owed to this reconcile for #1217. The reason the row's gate is
not recorded as a pass on `main` is restated as `## Now` gives it, which is that
the 96/96 was measured on the #1221 branch tree and no run against current `main`
exists.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…DING because I read a body written before its own job ran

The previous commit here wrote the A3 end-to-end token gate as `PENDING` and
said no GB10 96/96 existed. That was wrong, and the way it was wrong is worth
keeping: I searched `origin/main` and five `row/MODEL-NEMOTRON-H-*` branches,
found nothing, and then took #1221's PULL REQUEST BODY as the current state. Its
"Still owed: the sm_121a green-after re-run" was TRUE WHEN WRITTEN and stale by
the time I read it. A body is a snapshot of its author's knowledge at write time,
not a live field, and no amount of branch searching corrects for that, because
the artifact was never in git at all.

It is on the NAS. `/usr/local/nas_share/rc/nh1157/gate_fixed.out`:

    TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
    STRICT PASS

against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on
`nemotron-3.5-lightning-30b-nvfp4` at revision `29f2d1746d8f41e316523194b19018707749b1b1`.
Verified by reading the files, not by trusting the report of them.

It is the DEVICE leg. The binary is `libvllm 0.0.3+cuda` and `cfg.log` records
`fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` `ENABLED for [121a]`. The run logs
`Asynchronous scheduling is enabled (max_concurrent_batches=2)`, which is exactly
where `device_token_ids` is non-null; on the host queue it is always null and the
#1157 defect cannot arise. The decisive one is neither: the same binary on the
same checkpoint with ONLY `nemotron_h_device.cpp` reverted to the fix's parent
scores `4/24 (full rows=0, short rows=3)` and bails at 8 tokens. A host-leg run
would have been unmoved by that revert. The delta is the proof; the pass alone
is not.

So the row now says the gate PASSES and says whose it is. It belongs to #1221,
branch `row/MODEL-NEMOTRON-H-ABI-A2P-1157-fix` at `6e9e8955`, OPEN and
`CONFLICTING`. `main`'s last touch of `nemotron_h_device.cpp` is `a6df72777`
(A2-P), so `main` does not carry the fix and is not gated. A gate that passed and
a tree that is gated are different facts, and this record keeps them apart.

No performance number is claimed anywhere. `gate_fixed.out` carries 264.4s to
load and 327-343s per 32-token prompt; that is a correctness run whose `lm_head`
and 46 FP8 mamba projections still execute host-side. Both public rows say so in
the words "NOT a benchmark", so nothing can later lift them into a grid. One
config caveat also travels: `--gpu-memory-utilization 0.92` did not size the KV
pool, which fell back to 256 blocks (#83).

The row stays `PARTIAL`. `ACTIVE` still fails the `CLAIM-*` rule, and a gate that
passes off `main` is not a reason to move a lifecycle state.

`docs/FEATURES.md` moves in the same commit, because it has to. This commit
edits `.agents/model-matrix.md`, which `scripts/check-doc-checkpoint.py` reads as
a `feature_surface` change by path (`FEATURE_SURFACE_FILES`), and that class owes
`docs/FEATURES.md` in the SAME commit -- the checker walks a range one commit at
a time, so no later commit can pay the debt. The row for
`NemotronHForCausalLM` said nothing at all about the end-to-end token gate while
the matrix said it was PENDING, and it would have kept saying nothing now that
the gate passes. It now carries the same two facts the matrix does, in the
correctness column and in that column's own words: 96/96 `STRICT PASS` on GB10,
on #1221's open branch and not on `main`. The added text is 90 characters, which
leaves the cell at 192 of its 220-character cap and the row at 499 of 600, so no
other row pays for it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` gained #1221 while this branch was open, so `docs/BENCHMARKS.md` and
`docs/STATUS.md` conflicted on the NemotronH rows. Resolved by taking main's
rows as the base and correcting the one claim that main is now STALE on.

Main says "GB10 read 4/24 ... sm_121a re-run pending a lease" and "sm_121a
re-run pending". That was true when #1221 was written. The re-run has since
happened: the A3 gate PASSES on GB10 at 96/96, `STRICT PASS`, mode=decode,
against `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4`
revision `29f2d1746d8f41e316523194b19018707749b1b1`, with a red-before of 4/24
on the SAME binary and checkpoint with only `nemotron_h_device.cpp` reverted.
Artefacts: `/usr/local/nas_share/rc/nh1157/{gate_fixed,gate_red,cfg}.out`.
`cfg.log` shows `fp4-mma`, `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and
`fa2` all `ENABLED for [121a]`, so the build was not degraded, and both legs log
`Asynchronous scheduling is enabled`, which is the path where
`device_token_ids` is live and the defect lived.

The delta, not the pass, is the proof: on the host leg reverting the fix would
have changed nothing.

STILL NOT A BENCHMARK. The 264.4 s load and 43 405 MB peak are recorded as a
correctness run and labelled as such, because NVFP4 `lm_head` (A2-Q2b) and the
46 FP8 W8A8 mamba projections (A2-Q1, #940) still execute host-side. No
throughput, latency or memory ratio is claimed on any axis.

The index was reconciled by discard-and-reapply: main's file taken wholesale,
only rows whose key main lacks re-appended, main's file a strict PREFIX.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…ncile rested on expired (#1074)

FOLLOWING_AGENTS_PROTOCOL

This reconcile was written while [#1221](#1221)
was OPEN, and it moved the row `INVENTORIED` -> `PARTIAL` on the strength of an
A3 gate that read 96/96 `STRICT PASS` on GB10. #1221 MERGED on 2026-08-18 as
`0ea5d249f`, which is now `main`'s last touch of `nemotron_h_device.cpp`, so
every "blocked on #1221 landing" sentence in this branch became false.

What is recorded now is narrower than what the branch originally claimed, and
deliberately so. The A3 gate read 96/96 `STRICT PASS` (mode=decode) on GB10,
MEASURED ON THE TREE THAT LANDED AS `0ea5d249f`. That fix is on `main`. **No run
against current `main` exists**, because `main` has advanced many commits past
the measured tree. Naming the SHA a measurement belongs to is the whole point:
an evidence line that names a tree it was not measured on has cost this
repository before, and this change does not add another.

**The row therefore STAYS `INVENTORIED`.** The lifecycle move is left to whoever
re-runs the gate against `main`, and the derived counts in
`.agents/model-matrix.md` are restored to main's `53`/`324`/`22` with the
engaged-architecture row withdrawn. Measured rather than asserted: the row-state
histogram over all 373 keyed rows is byte-identical to main's, so this change
moves NO row's lifecycle state, and the count block is byte-identical to main's.
What survives from the original reconcile is the part that was never about
#1221: the row's `KERNEL-SSM-MAMBA` block was FALSE, and its corrected code and
test anchors stay, so a later state move has its contract already written.

`docs/BENCHMARKS.md` carried the key `NemotronH paged forward`
(`MODEL-NEMOTRON-H-ABI-A2P`) TWICE on this branch and once on main, because the
branch APPENDED where it should have UPDATED. The two rows contradicted each
other: main's said GB10 read 4/24 with an sm_121a re-run pending a lease, and
the branch's said GB10 read 96/96. The lease and the re-run happened, so main's
row is stale on its face. They are collapsed to ONE row on the key byte-identical
to main's, carrying the current projection. Nothing gates this:
`check-public-doc-tables.py` holds the page's size and shape and not its key
uniqueness, so it would have landed silently.

`docs/STATUS.md` gets a one-line in-place edit to the `Text generation` row's
"Open gate or limitation" cell, which is where #1491 routed this after it
reorganised the page by surface. That is the shape `af25bd251`, `5702d8f83` and
`d995c52f0` established for `Speculative decoding` and the diffusion row, and it
satisfies the `lifecycle` obligation without growing a shrink-only page: 1
insertion, 1 deletion, 98 lines and 27 table rows before and after. Model-level
detail is NOT reintroduced; it stays in
[`docs/models/nemotron-3-5-lightning.md`](../docs/models/nemotron-3-5-lightning.md),
whose "no token gate result exists" line remains true of `main`.

The obligation still fires with no lifecycle move because
`check-doc-checkpoint.py` adds the `lifecycle` class for a measurement change as
well as a state change, so `STATUS` and `BENCHMARKS` are both owed and both
written.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

… open, and this branch added that text (#1074)

The `## Owed` entry for #1217 was written while #1221 was open and it kept three
claims that are false at this head. It called #1221 "open and unmerged", although
#1221 merged on 2026-08-18 as `0ea5d249f`, which the previous commit's own
subject says. It said #1217 "is recorded NOWHERE in this tree", although the
merge `8a00afcbe` brought both records in: `.agents/issue-index.md:403` carries
the index row under the owning row `MODEL-NEMOTRON-H-ABI-A2P`, and
`nemotron-h-a2p-paged-forward.md:901` lists the issue under that spec's
`## 11. Owed`. That claim was true at `b626be75a` and is not true here. And it
named #1217 "the standing reason this row's end-to-end gate is not a pass", which
contradicts `## Now` in the same file, where the A3 gate reads 96/96
`STRICT PASS`.

`git diff 6b48edb 414a800` shows this branch ADDED the phrase, and
`origin/main` carries it zero times, so the stale text is this pull request's to
remove rather than an inherited defect.

The bullet now records where the record landed and cites both anchors, which
were re-derived at this head. The duplicate-key argument for withholding an index
row is kept as history and marked expired: appending a row now would create the
very duplicate that argument was written to avoid, because the row is already on
`main`. Nothing is owed to this reconcile for #1217. The reason the row's gate is
not recorded as a pass on `main` is restated as `## Now` gives it, which is that
the 96/96 was measured on the #1221 branch tree and no run against current `main`
exists.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…nded, and a file glob is why nobody saw it (#1081)

The NemotronH model-matrix row said the architecture was BLOCKED on
`KERNEL-SSM-MAMBA`
([#496](#496))
because "the Mamba2 SSD core is unported". That kernel landed on
2026-08-13.
The row also carried `☐ required` for its spec and `none` for its
evidence while
five specs were committed and the paged forward, the device MoE arm and
the ABI
driver had all landed. This change corrects the record. It does not move
the
row's lifecycle state.

FOLLOWING_AGENTS_PROTOCOL

## Why a directory listing hid the blocker

[#496](#496) landed its host
arm at
`47960a009` (W1) and its CUDA arm at `43a6c5518` (W2). The kernel is
`src/vt/cuda/cuda_mamba2_ssd.cuh` — a 692-line **header**, `#include`d
at
`cuda_gdn.cu:48` and registered as `kMamba2ChunkScan` at
`cuda_gdn.cu:6669`, not
a translation unit of its own. A `src/vt/*mamba*` FILE GLOB therefore
returns
nothing and reads as absence, and that glob is the search both the row's
text and
[#1074](#1074) rested on.
`nemotron_h.cpp:597` calls `vt::Mamba2ChunkScan` today, and
`nemotron_h.cpp:354`
calls `vt::MoeRelu2`. #496 stays open for its GENERIC `MambaSpec`
producer, which
this architecture does not need, because A1 made the runner read the
model's own
KV spec.

Worth carrying: **#496's own issue TITLE still asserts "Mamba2 SSD core
is
unported"**, so the false claim has a second live copy this change
cannot reach.

Three claims in the row were FALSE and are corrected: the
`KERNEL-SSM-MAMBA`
block, the non-gated `relu²` MoE "exists nowhere locally", and ModelOpt
`MIXED_PRECISION` loading "exists nowhere locally". One was TRUE and
stays: the
DeepSeek-style MTP head does not exist locally, and it remains owed as
W5.

## The row STAYS `INVENTORIED`

This is the part an earlier draft of this description got wrong, and the
correction is the point of the change. That draft moved the row
`INVENTORIED` ->
`PARTIAL` with the mark `🚧`, the rollup `INVENTORIED` 324->323 and
`PARTIAL`
22->23, and the two prose counts 53->54. **None of that is in this
diff.**

Measured on this head against `origin/main`:

- the row's lifecycle cell is `` `INVENTORIED` `` on both sides;
- the rollup table `.agents/model-matrix.md:76-86` is BYTE-IDENTICAL to
`main`'s
— `INVENTORIED` 324, `PARTIAL` 22, `ACTIVE` 10, `SPIKE` 9, `BLOCKED` 5,
`DONE`
  3, `READY` 3, `GATING` 1, total 377;
- `scripts/check-agent-record.py` derives `MODEL=377` and its model
inventory
`{rows: 324, memberships: 373, architectures: 356, targets: 310,
modules: 261}`
  unchanged.

The lifecycle move is left to whoever re-runs the gate against `main`.
What
survives is the part that was never about a state change: the row's
FALSE claims,
and its corrected code and test anchors.

## The A3 gate PASSES, and it belongs to a named tree

An earlier draft recorded this gate as PENDING, because it read
[#1221](#1221 pull request
BODY as the
current state. That body's "Still owed: the sm_121a green-after re-run"
was true
when written and stale by the time it was read. A body is a snapshot of
its
author's knowledge at write time, not a live field.

The artefact is on the NAS at `/usr/local/nas_share/rc/nh1157/`.
Verbatim from
`gate_fixed.out`:

```
TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
STRICT PASS
```

against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on
`nemotron-3.5-lightning-30b-nvfp4` at revision
`29f2d1746d8f41e316523194b19018707749b1b1`.

**It is the DEVICE leg.** The binary is `libvllm 0.0.3+cuda`, and
`cfg.log`
records `fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` as `ENABLED for
[121a]`. The
run logs `Asynchronous scheduling is enabled
(max_concurrent_batches=2)`, which
is precisely where `device_token_ids` is non-null; on the host queue it
is always
null and the [#1157](#1157)
defect
cannot arise. The decisive evidence is neither of those: the same binary
on the
same checkpoint with **only** `nemotron_h_device.cpp` reverted to the
fix's
parent scores `4/24 (full rows=0, short rows=3)` and bails at 8
generated tokens.
A host-leg run would have been unmoved by that revert. **The delta is
the proof;
the pass on its own is not.**

**The pass was measured on the tree that landed as `0ea5d249f`, and
there is no
run against current `main`.** #1221 MERGED on 2026-08-18 as `0ea5d249f`,
which is
`main`'s last touch of `nemotron_h_device.cpp`, so the repair the 96/96
depended
on is on `main`. That is not the same fact as a gated `main`: the 96/96
was taken
on that branch tree, `main` has advanced since, and nobody has re-run
it. Every
row here therefore names the SHA the measurement belongs to and records
no pass
on `main`. An earlier draft of this description instead called that
branch "OPEN
and `CONFLICTING`" with "the merge is owed", and quoted a shipped cell
as reading
"on #1221's OPEN branch, not on `main`". Neither is true here, and that
quoted
text ships nowhere: `grep -c "OPEN branch"` over `docs/` and
`.agents/model-matrix.md` is 0.

**No throughput, latency or memory figure is claimed anywhere in this
change.**
`gate_fixed.out` carries 264.4s to load and 327-343s per 32-token
prompt. That is
a correctness run whose `lm_head` and 46 FP8 mamba projections still
execute
host-side. Both public rows say "NOT a benchmark" in those words. One
config
caveat travels with the run: `--gpu-memory-utilization 0.92` did not
size the KV
pool, which fell back to 256 blocks
([#83](#83)).

## The stale `## Owed` bullet for #1217

The `## Owed` entry for
[#1217](#1217)
in `.agents/specs/nemotron-h-model.md` was written while #1221 was open,
and this
branch ADDED it: `origin/main` carries the phrase "open and unmerged"
zero times.
It was false three ways at its own head, and the last commit repairs it.

| the bullet said | this head has |
|---|---|
| #1221 "is open and unmerged" | merged 2026-08-18 as `0ea5d249f` |
| #1217 "is recorded NOWHERE in this tree" |
`.agents/issue-index.md:404` and `nemotron-h-a2p-paged-forward.md:901` |
| "the standing reason this row's end-to-end gate is not a pass" | the
same file's `## Now` records 96/96 `STRICT PASS` |

The "recorded nowhere" claim WAS true at `b626be75a`, which is
verifiable —
`git show b626be7:.agents/issue-index.md | grep -c 1217` is 0. The
merge
`8a00afcbe` brought both records in. Both anchors were re-derived at
this head:
`issue-index.md:404` carries the row under the owning row
`MODEL-NEMOTRON-H-ABI-A2P`, and `nemotron-h-a2p-paged-forward.md:901` is
the
first bullet under that spec's `## 11. Owed`.

The duplicate-key argument for withholding an index row is kept as
history and
marked expired. Appending a row now would create the very duplicate that
argument
was written to avoid, because the row is already on `main`. **Nothing is
owed to
this reconcile for #1217.** The issue stays open as a seam defect owned
by
`MODEL-NEMOTRON-H-ABI-A2P`.

## Two defects found and fixed in flow

`docs/BENCHMARKS.md` carried the key `NemotronH paged forward` **twice**
on this
branch and once on `main`, because the branch APPENDED where it should
have
UPDATED, and the two rows contradicted each other (4/24 with a re-run
pending,
versus 96/96). They are collapsed to one row on the key. `grep -c` for
that key
is now 1 here and 1 on `main`. Nothing gates this:
`scripts/check-public-doc-tables.py` holds the page's size and shape,
not its key
uniqueness. The row's position moved when the duplicate was collapsed
and it now
follows `Record-anchor ratchet`; the key and content are correct, and
reordering a
file that other pull requests edit was not worth the conflict.

`docs/STATUS.md` was restructured under this branch by
[#1491](#1491), which reorganised
it by
surface and deleted the per-model table, so the merge conflicted for
real. The
page is resolved to `main`'s whole and the record is re-expressed as a
one-line
in-place edit to the `Text generation` row.

## A latent defect this branch introduced, and the replay that removes
it

The row's corrected cell-6 prose linked #496's spec inline as
`[#496](…) ([spec](specs/mamba2-ssd.md))`.
`scripts/check-doc-checkpoint.py`'s
`spec_for_row` takes the **first** `specs/*.md` link anywhere in the row
line, so
that parenthetical shadowed the row's own spec: `MODEL-TEXT-nemotron-h`
resolved
to `mamba2-ssd.md` instead of `nemotron-h-model.md`. On `origin/main`
the same row
resolves correctly to `nemotron-h-model.md`, so this branch introduced
it.

**This is not a gate artifact, and the squash would not have absorbed
it.** The
shadowing lands and persists: every future lifecycle move of this row
would
demand a `## Now` from `mamba2-ssd.md`, a spec whose author has no
reason to
touch it, with nothing pointing back at the row that caused it. The fix
is
correct on its own merits; the gate going red was how it was found, not
why it
is being made.

It surfaced because #1577, #1609 and #1621 landed on `main` under this
branch and
gave the checker sight of the states it had been blind to — `PARTIAL`
joined
`STATES`, `INVENTORIED` joined the new `RECORD_STATES`. `414a8002b`
moves this row
`PARTIAL` -> `INVENTORIED`, which is a real move the old checker could
not see.

`errors_for` is strictly per-commit (`commits_in_range` ->
`errors_for(paths,
parent, commit)`), so no later commit can pay the debt. The 27 branch
commits were
therefore replayed with `git filter-branch --index-filter`, scoped
`rewrite-1081 --not origin/main` so that **no commit of `main` was
rewritten**.

Proof, not assertion:

- `origin/main` at replay time (`e100e64e1`) was NOT rewritten: it still
resolves
  as itself and remains an ancestor of the head.
- `git diff <old-head> <new-head>` is **one file, one line** — the
parenthetical.
- All 27 old/new pairs compared: the only path that differs is
  `.agents/model-matrix.md`, and author name, author email, author date,
committer name, committer email, committer date and subject are
byte-identical
on every pair. The five commits that never contained the string kept
their
  original SHAs, because the filter is a no-op on them.
- The shadowing string now occurs **0** times across all 27 commits, and
`mamba2-ssd.md` is still linked from the row's `Spike/spec` cell, so no
  reference is lost.

## Which projections this change owes, and why

`scripts/check-doc-checkpoint.py` was instrumented on this exact diff
rather than
reasoned about. `classify()` returns classes `{'feature_surface'}` and
`lifecycle_moves` `[]`, and `MEASUREMENT_RECORDS` is
`('.agents/benchmark-record.md',)`, which this change does not touch.

An earlier revision of this description said the checker's `STATES`
tuple
"contains neither `INVENTORIED` nor `PARTIAL`, so it is structurally
blind to
this row's state". **That was true when measured and is false at this
head**, and
the conclusion survives while the reason inverts. After #1577, #1609 and
#1621 the
gate SEES both states and reports no lifecycle move across `origin/main`
-> head,
because the row genuinely does not move: `INVENTORIED` on both sides. A
gate that
looks and finds nothing is a stronger result than a gate that cannot
look.

`docs/STATUS.md`, `docs/BENCHMARKS.md` and the spec's `## Now` are
therefore
written because AGENTS.md `## Public documents` owes them, **not**
because a gate
asked. `docs/FEATURES.md` IS touched, at `:151`: the
`NemotronHForCausalLM` row
said nothing about the end-to-end token gate before, and its correctness
column
now reads "A3 e2e gate 96/96 `STRICT PASS` on GB10 at `0ea5d249f`
(#1221); NO run
against current `main`".

The `docs/STATUS.md` edit takes `oversized_cells` from 2 to 3 against a
ratchet of
44, and `long_paragraphs` stays 0 against 75. No ratchet is raised.

## `.agents/issue-index.md`

GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so
this
branch has repeatedly discarded the union auto-merge, taken `main`'s
file whole,
and re-appended only the rows whose KEY `main` lacks: `#1074` and
`#1080`. `main`
already carries a `#1068` row, so that one is not re-appended.

Measured on this head against `origin/main`: **565 rows, of which
`main`'s 563 are
all present, in order, byte-identical; 0 removed; exactly 2 added.**
`scripts/check-issue-index-append-only.py` exits 0. The file is not a
strict BYTE
prefix of `main`'s, because `main`'s newer rows follow this branch's
two; the
append-only rule is about rows rather than bytes.

`#1080` is re-measured rather than carried: after this change repairs
`nemotron-h-model.md`'s `## 7. Now` to `## Now`, **15** specs still
write the
numbered spelling, up from the twelve #1080 was filed against. A growing
population is the argument for the checker-semantics fix over a rename
sweep, and
both need their own spec and a red-before test. It is listed under `##
Owed` in
`nemotron-h-model.md`.

## Reconciling with #1415, which landed on both Nemotron rows

`main` gained the A2-Q2b device `lm_head` (#1415) while this branch was
open, and
it edited the same two pages. These were real content conflicts, not the
union-driver artifact the index produces. Both were resolved BY KEY, and
neither
side is discarded.

`docs/BENCHMARKS.md` was POSITIONAL rather than a same-key dispute. This
branch had
already moved `NemotronH paged forward` below `Record-anchor ratchet`
when it
collapsed that key's duplicate, and #1415 inserted a NEW key at the old
offset.
Three-way: `main` did not touch `NemotronH paged forward` since the
merge base, so
this branch's version stands, and `NemotronH host re-expansion / decode
token` is a
main-only key taken verbatim. **Each key occurs exactly once.**

**The reflex resolution here is wrong, and nothing would have caught
it.** Taking
`main`'s version of the conflicted region — the ordinary move when the
other side
looks newer — would have re-created the exact duplicate this pull
request exists
to remove, because our copy of that key lives further down the file and
the
conflicted region is not where the key now is. It would also have looked
clean:
`scripts/check-public-doc-tables.py` holds these pages' size and shape
and does
NOT test key uniqueness, which is
[#1543](#1543). A gate could
not have
caught the regression. Only decomposing the conflict BY KEY, and asking
the
three-way question "did the other side actually touch this key",
separated a
position collision from a content disagreement. Anyone resolving a
conflict in a
keyed projection in this tree should assume the same trap.

`docs/FEATURES.md` is the same key, `NemotronHForCausalLM`, changed by
both sides
**in disjoint cells**. Cell 3 changed only here, adding the A3 gate
result. Cell 4
changed only on `main`, moving device `lm_head` out of the owed list and
recording
it UNMEASURED. The row is composed from both, so nothing is adjudicated
and nothing
is dropped.

**#1415 landing does not gate this row, and this merge does not upgrade
the
claim.** Cell 3 still reads "NO run against current `main`". A device
`lm_head`
that reaches the device is not a run of this row's end-to-end token
gate. FP8
mamba stays owed, because A2-Q1 (#1289) is still DRAFT.

Verified rather than trusted: every other key in both pages is
byte-identical to
`main`; `docs/STATUS.md` auto-merged with `Text generation` as its only
differing
key, no key added or lost; the model-matrix rollup is byte-identical and
the row is
still `INVENTORIED` on both sides.

## Gates

Run on `1b741c66baec6c9660aa80af3a82c0f5eca24dd9`, this pull request's
head, with
`origin/main` at `e7f38e0c01ff1d85d18ef64eadd98996743ca061`.

`scripts/agent-preflight.sh --fail-on-skip` reports **ZERO gates
SKIPPED**, and
`doc-checkpoint range` is `ok`. `RANGE_COUNT` is **28**. Both trailer
gates were
proven to READ that range by mutation rather than read off a green line:
a
detached scratch commit with no trailer block makes
`check-commit-trailers.py`
exit 1 naming `3aeb80e9b769`, and a scratch commit whose subject ends in
a period
makes `check-commit-style.py` exit 1 naming `487921ac602c`. The tree was
restored
to `1b741c66b` after each, verified by SHA and a clean `git status`.

`doc-checkpoint` is green for the right reason, and that was falsified
too.
`check-doc-checkpoint.py --commit 4dff515` (the replayed `414a8002b`)
exits 0
while the `PARTIAL` -> `INVENTORIED` move is still present; renaming
that spec's
`## Now` heading in a scratch amend takes it to exit 1 with
"`.agents/specs/nemotron-h-model.md` has no `## Now` section". So the
gate passes
because the obligation is satisfied, not because the move vanished.

One trap worth recording: the FIRST attempt at that mutation read as
passing. It
was run from a tree checked out at `4dff5158b`, which predates the merge
that
brought #1577 in, so it executed the OLD blind checker against the new
commit.
The instrument was the stale thing, not the result.

Bare, on the same head, each exit 0:

```
check-doc-checkpoint.py --base e7f38e0 --head HEAD   OK
check-public-doc-tables.py                             OK
check-supported-models.py                              OK: exactly the 40 registered architectures
check-agent-record.py                                  ENGINE=169 MODEL=377 QUANT=84 KERNEL=57 BACKEND=85 ANCHOR-ROT=37
check-issue-index-append-only.py                       OK: issue index append-only
check-symbol-anchors.py                                OK every symbol anchor still finds what it names
check-now-current.py                                   OK
```

`scripts/agent-preflight.sh` exits 1 on `test_cpu_x86_llamacpp_floor`
ALONE. That
is [#618](#618) and not a
finding: the
harness reported `waiting for quiet: 15s busy=109% builders=0
load=29.97`. The
test reads `scripts/cpu-x86-llamacpp-floor.sh` and
`docs/bench-evidence/cpu-x86-llamacpp-20260811.md`, and this change
touches
neither — it touches six record files and **zero code files**.

## Inherited, not introduced

`windows-msvc-cpu` and `windows-msvc-vulkan` fail on every pull request
and have
no `main` baseline
([#584](#584),
[#968](#968)). This change has
zero code
files and cannot reach a compiler.

Closes #1074

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…e 0.001392x, load 2.12x faster, and the GPU idle for 93.7% of our decode (#1250) (#1251)

There was no Nemotron number on any axis. `docs/BENCHMARKS.md` carried
`MODEL-NEMOTRON-H-ABI-A2P` as "No speed number, by the unit's own rule".
This
branch takes the first ones, against the pinned vLLM oracle, which
completed a
model run inside an `rc` lease for the first time.

FOLLOWING_AGENTS_PROTOCOL

## The numbers

Both legs on `dgx:gpu0` through `rc run`, same box, same boot id, same
checkpoint directory, same three pre-tokenized prompts, same 32 greedy
tokens
with `ignore_eos`, batch 1 sequential, two legs over one engine load on
each
side. Tree `5325b7b970b67f97a77834e907fc34fb2990b71e`, which contains
`0ea5d249f` (#1221). **`enforce_eager=False` throughout: CUDA graphs
were on and
were never disabled.**

| axis | ours | pinned vLLM | ratio |
|---|---|---|---|
| per output token, warm (n=5) | 10.3194 s | 0.014369 s | **718.2x
slower** |
| output throughput, batch 1 | 0.09691 tok/s | 69.595 tok/s |
**0.001392x** |
| engine load | 280.9 s | 596.3 s | **0.4711x, we are 2.12x FASTER** |
| peak host memory | 44,616 MB | 70,974 MB | 0.629x raw, not
like-for-like |
| KV pool | 8192 tokens | 644,096 tokens | 78.6x, could NOT be matched |

**Every timing leg is a gated leg.** Ours read `TOKEN MATCH: 96/96 over
3
prompt(s) (full rows=3, short rows=0, mode=decode)` on BOTH legs,
`STRICT PASS`,
192 of 192 tokens, so no number here comes from a configuration whose
tokens were
not compared in the same process. The same-binary A/B over one load is
1.0016
warm and the five warm prompts spread 0.245%.

The build was not degraded: CUDA 13.3.73, `CFG_RC=0`, `BUILD_RC=0`, zero
compile
errors, and `fp4-mma` / `cutlass-nvfp4` / `cutlass-fp8` / `marlin-nvfp4`
/ `fa2`
each `ENABLED for [121a]`, counted as `FEATURE_LINES_SEEN=5
DEGRADED_FEATURE_LINES=0` rather than eyeballed. The box was idle:
`--query-compute-apps` empty, 4,892 of 122,502 MB used.

## Two refusals that travel with the ratios

They are recorded beside the numbers rather than netted out of them.

**The clock gate refuses this pair.** `gpu_clock_state compare` exits 1
with six
reasons: ours 6.31% busy and vLLM 31.05% busy against a 50% floor, vLLM
spread
5.14% against a 5.0% ceiling, ours throttled `SwPowerCap`, persistence
`Disabled` on both. Same boot id, both medians 2411 MHz,
`median_offset_pct`
0.0. What the refusal cannot do is explain the result: the tool's own
recorded
basis is 0.7548 points of kernel time per point of clock, and the gap is
718x.

**The KV pools could not be matched.** vLLM's hybrid allocator resolves
its own
block size -- 512 here, 4192 at other settings -- against our 32, so a
token
capacity match is the most that was ever available and this run did not
get one.
The memory axis is therefore reported raw, and the straggler still
resident in
the oracle's window (PID 40514, 22,986 MiB, the EngineCore this row's
own
watchdog killed) is named rather than subtracted.

## What names the bottleneck, and why it is not a ceiling

**`nvidia-smi` reported GPU utilization 0% in 2,019 of 2,155 samples of
our
window. The GPU was busy in 6.31% of it.** That is the driver's own
answer, not
a reading of the source, and the clock helper's refusal to attribute
such a
window IS the finding: this decode is not GPU work.

It matches the mechanism exactly. Of 52 layers, 6 are attention and stay
on the
device, 23 MoE layers run on the device through the NVFP4 Marlin arm,
and 23
Mamba2 layers bounce -- the normed hidden is downloaded, the mixer runs
on the
CPU queue, the result is uploaded -- once per layer per token; then
`NemotronHHostLmHead` projects on the host because
`src/vllm/model_executor/models/nemotron_h.cpp::NemotronHHostLmHead`
refuses the NVFP4 `lm_head` on a non-CPU queue.

That citation is a SYMBOL rather than a line, here and in the spec and
in the
index row, and the change is a repair. All three said
`nemotron_h.cpp:1031-1034`;
the refusal is at `:1036` now and those lines hold `H`, `V` and two size
checks.
`scripts/check-symbol-anchors.py` says in its own docstring that it
"does not
verify LINE citations. There is no honest way to", so nothing reported
it. The
index row is the one that had to be fixed before this merges: that file
is
append-only, a landed row may never be edited, and the same checker
lists it in
`FROZEN_FILES` because "a stale anchor inside one is unrepairable by
construction".

## The lever is confirmed, and still ungated

A three-leg discriminator moved those 23 layers on-device and reported
both
halves of this row's prediction rather than speed alone:

| leg | arm | token gate | warm s/token | GPU busy |
|---|---|---|---|---|
| `a3_hostmamba` | host bounce, the SHIPPED default | **96/96 `STRICT
PASS`** | 10.1502 | 7.86% |
| `a3_off` | device `Mamba2ChunkScan` | **95/96 `DIVERGENCE`** | 1.3898
| 10.18% |
| `a3_on` | device `Mamba2StateUpdate` | **95/96 `DIVERGENCE`** | 1.3947
| 10.27% |

**7.28x** warm, on the same cold-prompt exclusion used throughout, and
the busy
fraction rises as predicted. (A 6.64x figure for the same pair includes
the cold
prompt on both sides.)

**It is not a parity number.** Both device legs read 95/96, and
AGENTS.md
establishes the token gate BEFORE a performance result is accepted, so
718.2x
stays the gated figure and the device arm's ~97x-vs-oracle is carried as
an
ungated projection. #1289 is DRAFT for exactly this. And no ceiling is
in sight:
at ~10.2% busy the decode is still ~90% GPU-idle, so A2-Q1 banks 7.28x
without
closing the gap, and A2-Q2b's `lm_head` owes the same test.

**#940 was never the live pointer** -- it has been closed since
2026-08-16. The
live tracker is #1289, blocked by #1388. Corrected everywhere this
branch cited
it; the two pre-existing #940 rows in the issue index belong to other
rows and
are untouched.

**Two findings went to #1388**, which had neither. Its "arch- or
host-specific,
not arm-specific" conclusion came from two DEVICE arms plus a passing
Thor, with
no host-arm leg on GB10; there are now three such legs and all pass
96/96 on the
same binary and box, so the divergence does track the arm. And the
diverging row
is prompt 2 -- which is also the row where the pinned oracle failed to
reproduce
its OWN committed golden (`matched=26` of 32) once its resolved
`block_size`
moved to 512. That is evidence for the benign-near-tie branch and
against a
wrong recurrent carry.

## The keyed records were rebuilt twice, and the second conflict was
POSITIONAL

An automatic three-way merge on this branch had damaged four keyed
records: it
duplicated the `NemotronH paged forward` and `Record-anchor ratchet`
rows in
`docs/BENCHMARKS.md`, duplicated this row's `#1250` index row, and
DROPPED
`ENG-EXPERT-STREAM-DEVICE`'s row from `docs/STATUS.md` and its section
from
`.agents/benchmark-record.md`.

Re-basing on current `main` produced the same class of defect from the
opposite
direction, and nothing would have caught it. The base carried
`**NemotronH paged forward**` directly after `**Developer/row
protocol**`, and
this branch edited it there. Main MOVED that row two positions down,
past
`**Record-anchor ratchet**`, and inserted a DIFFERENT key,
`**NemotronH host re-expansion / decode token**` (`A2-Q2b`), at the
offset the
move vacated. Git therefore conflicted this branch's edit against a row
it has
nothing to do with, and auto-merged main's relocated copy in BELOW the
conflict
as an addition. The conflicted tree held TWO `**NemotronH paged
forward**` rows.
Taking either side wholesale lands a duplicate key, and
`scripts/check-public-doc-tables.py` does not test key uniqueness
(#1543), so it
would have landed silently. This is the #1081 shape again.

Every one was rebuilt the way AGENTS.md prescribes -- take main's
complete
version, re-apply the scoped edit, verify unrelated keys byte-for-byte.
Each
public doc now differs from main by exactly one line, the index carries
main's
rows unchanged with two rows added and no duplicate keys, and the record
is
main's file with this row's one 245-line section inserted and nothing
removed.

"Strict prefix" is the wrong word for the index and it is worth being
exact
about, because the two claims are not the same guarantee. The two rows
do NOT
land at the end. Main's own later rows follow them, which is the
ordinary result
of two branches appending concurrently.

No offset or trailing-line count is quoted here, deliberately. That
number moves
with every merge of `main` -- it read 118 one base ago and 141 at this
head --
and `squash_merge_commit_message = PR_BODY` makes this description an
immutable
commit message, so a coordinate into a moving file is the same defect
here as a
`file:line` anchor is in a record. What IS true is the property the
append-only
rule actually needs, and it is measured at the head this pull request
pushes:
main's lines are an exact ordered subsequence of this tree's, the diff
is a
SINGLE insert opcode, nothing is removed, and no issue key appears
twice.
Main had also edited that scoreboard row's own cells, so the rebuilt row
keeps
this branch's measurement AND main's provenance caveat that the branch's
text
dropped: the 96/96 was read on `5325b7b97`, not on current `main`.

Two repairs to this branch's own `.agents/benchmark-record.md` section
ride
along, because that file is append-only too and this is the last cheap
moment
for either. Its section header carried its dated parenthetical TWICE; of
the 378
`## ` headers in that file it was the only one shaped that way, so it is
a slip
and not a convention, and it is the line a reader greps. And the section
carried
2026-08-18 while sitting between two 2026-08-19 sections; it now sits
below both,
so the file's newest-first order holds. Same 245 lines, different place.

`docs/STATUS.md` needed a different answer. #1491 rewrote that page from
65 rows
to 27, organised by surface rather than by model, with model detail
routed to
`docs/models/`, so the per-model `Nemotron-3.5-Lightning-30B-A3B` row
this branch
edited NO LONGER EXISTS there and a by-key re-apply would assert on a
missing
key. The speed result goes into the existing `Text generation` row's
open-gate
cell instead, one line changed in place, and the page keeps main's 27
rows.

## Also here

**`--num-blocks`, `--kv-cache-memory` and `--repeat` on
`nemotron-h-gen`.** The
driver could not state its own KV budget and inherited the 256-block
fallback
the engine takes when `gpu_memory_utilization` is accepted and does not
size the
pool (#83); both legs of the A3 gate logged that warning, and this run
does not.
`--repeat` runs the battery again over one load, and the verdict reads
grand
totals, so a battery that passed once and diverged on the second leg
cannot
report a pass.

**#1253**, filed and fixed in this flow. The oracle could not run a
model in a
lease for want of `python3-dev` -- Triton compiles `cuda_utils.c` at
runtime and
the failure surfaces four frames up as `Engine core initialization
failed ...
Failed core proc(s): {}`, an empty proc set naming nothing. And `nvcc
--version`
is not the toolkit postcondition: the worker ships a partial CUDA 13.0
whose
compiler answers while cuBLAS dev is absent, so CMake printed all five
feature
lines and then failed with `Target "vllm" links to CUDA::cublasLt but
the target
was not found`. #1185's recorded hazard for the oracle -- a reboot of
the box in
the step after `torch.compile` -- did NOT occur; the watchdog held.

Closes #1250.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>